Skip to content

feat(sleep): prototype community rules with local empirical gating - #265

Open
阿东玩AI (adongwanai) wants to merge 1 commit into
microsoft:mainfrom
adongwanai:feat/community-rule-manifest
Open

feat(sleep): prototype community rules with local empirical gating#265
阿东玩AI (adongwanai) wants to merge 1 commit into
microsoft:mainfrom
adongwanai:feat/community-rule-manifest

Conversation

@adongwanai

Copy link
Copy Markdown

Summary

  • add a versioned skillopt.community-rules v1 manifest for distilled rules,
    provenance-free rationales, licenses, and aggregate observed effects
  • export accepted skill/add edits without copying transcripts, tasks, memory,
    responses, or evidence logs
  • require explicit manual review before importing external rules
  • evaluate every imported rule independently on the importer's local val
    split, requiring strict score improvement with zero task-level regressions
  • stage accepted rules through the existing review/adopt boundary

Safety model

  • the manifest schema rejects unknown fields and has no transcript field
  • publisher-reported effects are informational and never affect local gating
  • observed effects are explicitly scoped to the complete candidate set
  • imported rules are never automatically adopted
  • train and test tasks remain outside the import decision

Non-goals

  • hosted registry, moderation, or automatic publication
  • transcript anonymization or transcript exchange
  • automatic discovery or adoption of remote rules

Validation

  • python3.12 -m pytest -q
    • 1502 passed, 11 skipped
  • uvx ruff check skillopt_sleep/community.py skillopt_sleep/__main__.py tests/test_community_rules.py
  • mkdocs build --strict

Addresses #156.

@adongwanai

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@Yif-Yang

Copy link
Copy Markdown
Contributor

Reviewed b3f9ac361201 as an isolated test merge onto main at 79124b37e9a6.

The incremental local gate and explicit review/stage/adopt boundary are useful: each later rule is evaluated against the already-accepted skill, and publisher-reported effects do not decide acceptance. Local validation passes: 1508 passed, 9 skipped, plus mkdocs build --strict.

I have not reproduced a specific runtime defect in this revision. However, this introduces a new community-rule exchange workflow, not just a maintenance fix. Please make the acceptance/value case concrete before we decide to take that scope:

  1. Add an end-to-end export -> review -> import -> explicit adopt regression. Assert that live skill/memory bytes are unchanged before adoption, a rejected import supplies no adoptable modification, and a successful explicit adoption retains the existing backup/baseline checks. The current positive CLI test reads the staged proposal but does not exercise that final boundary.
  2. Add a multi-rule gate regression: accept one rule, reject a later interacting/regressing rule, and then evaluate the next rule against the accepted state. Check both the final proposal and the per-task trial records; the current single-rule examples do not cover this composition.
  3. Include one small, sanitized usage receipt showing where sharing a rule is actually useful: local baseline/candidate scores, sample count, accepted/rejected rules, and replay cost. Keep train/test outside the import decision. MockBackend tests establish plumbing, not empirical transfer benefit; please distinguish those claims. Do not upload transcripts or private task contents, and retain the documented manual review of free-text rules/rationales before sharing.

This is an acceptance/evidence request, not a claim that the prototype contains a confirmed vulnerability. Official CI for this exact head is still awaiting maintainer approval; local green tests are not a substitute for that gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants